"Method, Terminal, Browser Application, and Mark-Up Language for Multimodal Interaction between a User and a Terminal" 

Jurgen SIENEL et al. - Attorney Docket No. Q760007 
Sheet ) of 4 



61 62 63 6 

4 J i4± 


4 65 


66 67 

mm 


50 

-ft 

. 35*. 



40 



41 



42 



43 



41ai^> 
==i36 



42a 




43a 



44 



38 



44a 



30 



7 




31 




9 



7 



Figure 1 




Figure 2 



'Method, Terminal, Browser Application, and Mark-Up Language for Multimodal Interaction between a User and a Terminal" 

Jurgen SIENEL, et al. - Attorney Docket No. Q760007 
Sheet > of 4 



Input / 
output 



100 



Evaluating time 
information 



101 



102 



104 



YES 



NO 



Conflicting / 
co-operative 
events? 



105 



Generating of a 
conflict corrected 
input/output 
according 
hierarchical rules 



106 ' ^ 


r 


Generating of a 
processed 
input /output 
according co- 
operative rules 



103 



processing 
input / output 



Figure 3 



"Method, Terminal, Browser Application, and Mark-Up Language for Multimodal Interaction between a User and a Terminal' 

Jurgen S1ENEL, et al. - Attorney Docket No. Q760007 
Sheet 4_ of 4 



61 



62 



63 



64 



65 



66 



67 



50 



r 



Figure 4 



45a 


45b ^ 


46a 


46b ^ 


47a 


4/b r. 


48a 


48b ^ 



« 

an 




£ 




7 



40 



20 



45c 



46c 



47c 



48c 



J 



61 71 



74 













12 




73 



y Z5 



46a 



76 



77 



46 



46c 



81 



80 



- 79 



78 



Pre-Processing: Front End 46a al 
Terminal 2 



Final processing: Back End 46c at 
Server 3 



Figure 5 



